This article originally appeared in The Bar Examiner print edition, Winter 2017-2018 (Vol. 86, No. 4), pp 4–6.
By Judith A. Gundersen
Greetings from Madison with our aptly titled Winter issue of the retooled Bar Examiner. Amy Kittleson, NCBE’s graphic designer—working with Claire Guback, the Bar Examiner’s editor—has come up with an engaging, fresh look for the magazine, and in this issue we begin to introduce some new content that we hope you will enjoy (see the new Facts & Figures page). Also beginning with this issue, we include any noteworthy news (such as litigation cases of interest, previously reported in the Litigation Update) in the News and Events section, with concise headline highlights and links for more information. There will be more changes as 2018 progresses, so keep reading! We also seek to engage you, our readers, by soliciting your testing questions for upcoming Testing Columns (see the end of this issue’s Q&A Testing Column). In fact, we welcome your suggestions and feedback regarding any content of this publication, which you may communicate to Claire at contact_editor@ncbex.org.
With the July 2017 exam results in the rearview mirror (the last jurisdiction released its results in mid-November), most jurisdictions saw an improvement in pass rates—some in double digits. And as I wrote in the September issue, the July 2017 MBE mean was up 1.4 points over the July 2016 mean, which showed a modest uptick from the July 2015 mean (see the new Facts & Figures page for a graphical depiction of this upward trend).
You might wonder why it can take until November for examinees to find out if they passed or failed the bar exam. Good question. The truth is that a lot has to happen behind the scenes at NCBE and in the jurisdictions before those results are ready to be released. First, NCBE has to score the MBE. Completing MBE scoring and disseminating the results to jurisdictions takes our Testing and Research and Test Operations Departments staff about a month. Over a week of that time is simply spent retrieving answer sheets—approximately 100,000 of them! (There are two answer sheets for each examinee—one for the morning testing session, and one for the afternoon testing session.) Every answer sheet is scanned at NCBE, sent to an off-site Scantron machine for answer data collection, and then sent back to our office. Once answer data are retrieved and entered into our scoring database, our psychometricians analyze the statistical performance of all items, after which the exam is equated. Equating—a process undertaken in every standardized test—ensures that slight variations in exam difficulty are accounted for; that is, an examinee is not penalized for getting a more difficult version of the exam or rewarded for getting an easier version. (See this issue’s Testing Column for a further explanation of equating.) After equating, MBE scaled scores are assigned to examinees. After NCBE Testing and Research Department staff members have checked and rechecked scores, the scores are sent to the jurisdictions. And all along the way, there are redundancies and parallel processes built into the scoring, scaling, and quality-control procedures to ensure 100% accuracy. Just about the entire NCBE Testing and Research and Test Operations Departments are fully deployed in the scoring and score reporting process from the end of July to late August in order to get the MBE scores sent to jurisdictions.
By the time jurisdictions receive their examinees’ MBE scores, most of them have been preparing for or have already started grading the written portions of their exams. (All jurisdictions, including those that use the Multistate Essay Examination [MEE] and the Multistate Performance Test [MPT], grade their own examinees’ papers.) Before grading can commence, jurisdictions undertake grader training and calibration. Calibration is the process by which graders develop coherent and identifiable grading standards so that grading is consistent throughout the grading process across answers and across multiple graders. In the case of multiple graders per question (a process used in large jurisdictions), it shouldn’t matter to an examinee if the examinee’s paper is graded by Grader A or by Grader B. Likewise, even where there is only one grader per question, it shouldn’t matter to an examinee if the examinee’s paper is graded first or last. Jurisdictions that use the MEE and the MPT (which includes all Uniform Bar Examination [UBE] jurisdictions) receive grading materials and training from NCBE at the Grading Workshop offered by NCBE after each exam administration. Individual workshop sessions are held for each question and are led by members of NCBE’s MEE and MPT Drafting Committees.
Graders then go about the task of blind-grading papers. Depending on how many papers a given grader is assigned, the grading process can take weeks (think jurisdictions with thousands of examinees). Grades are reported to jurisdiction staff members, who carefully check and record them. The written components are weighted according to the jurisdiction’s rules (in UBE jurisdictions, for example, the MEE is weighted 30%, and the MPT is weighted 20%, of the total exam score). Properly weighted, the written components are combined, and the total raw score of the written components is scaled to the MBE, which, in most jurisdictions, is weighted 50%. (Most, but not all, jurisdictions scale their written component grades to the MBE.) Scaling is a statistical process that adjusts the raw combined written scores for a given jurisdiction so that they collectively align with that jurisdiction’s distribution of MBE scaled scores. (See this issue’s Testing Column for a further explanation of essay scaling.) This process allows the written portion of the exam, which changes with every administration but cannot be equated like the MBE, to capitalize on the stability of the MBE, thereby accounting for any differences in question difficulty or grader stringency. Once scaling is completed, all results are checked and rechecked. Finally, results are ready to be released to examinees.
Whew. It’s a little easier to understand why all this takes a while, especially when it has to be done exactly right—and done exactly right for, in some cases, thousands of examinees.
As the July exam fades and the February exam looms, we are busy with new initiatives aimed at studying the future of the bar exam, exploring the migration of the MPRE to computer-based testing (CBT), and rolling out our new Investigations electronic application (for applicants requesting character and fitness investigations from NCBE, which NCBE conducts for participating jurisdictions). And, as you can see from the News and Events section of the magazine, we held two educational events in Madison this past fall, one for new bar admission administrators, and the other a training session for writing multiple-choice questions led by the chairs of our Civil Procedure and Contracts MBE Drafting Committees. The latter was a new educational event, and NCBE staff members who were involved really enjoyed meeting the nearly 40 faculty members in attendance who teach MBE subjects.
By the time this issue reaches you, we will have said good-bye to our cherished colleague Myra Andreassen, NCBE’s Manager of Operations for 20 years, who retired in early January 2018. We thank Myra for her years of maintaining and caring for our NCBE headquarters, as well as the support she has provided to NCBE’s Board of Trustees and the Council of Bar Admission Administrators (CBAA) throughout the years. (Oh, yeah, and she has also kept a full supply of candy at the front desk for those 20 years!) We wish Myra the best in her retirement as she enjoys time with her grandchildren and finds more time for traveling.
We will also have said good-bye to three good friends in the CBAA community: Gayle Murphy, Lee Ann Ward, and Fran Angell. Gayle, the Senior Director of Admissions for the State Bar of California Office of Bar Admissions, is retiring after 37 years with the Office of Admissions. Lee Ann, the Director of Bar Admissions for the Supreme Court of Ohio Office of Bar Admissions, is retiring after 21 years with the Office of Bar Admissions. Both were leaders in the bar admissions community, each having served as chair of the CBAA. Fran, the Attorney Admissions Administrator for the Kansas Board of Law Examiners, is leaving after 18 years. All three have been sources of wisdom, good humor, and expertise as members of the CBAA. There’s nothing the three of them haven’t seen in their years administering the bar exam. We wish them all the best.
And there’s plenty of news on the UBE front: the U.S. Virgin Islands adopted the UBE effective with the July 2017 exam, and North Carolina and Maryland recently announced adoption of the UBE in their states. North Carolina’s first administration of the UBE will be in February 2019, and Maryland’s first administration will be no sooner than July 2019. Our map of UBE jurisdictions is rapidly filling, with the total now at 30, as shown on our Facts & Figures page.
Finally, please take a moment to read, at the end of this issue, Chief Justice Mark Cady’s eloquent tribute to Dave Ewert, the late Assistant Director for Admissions for the Iowa Office of Professional Regulation, who passed away suddenly this past September.
I hope that you and your families had a wonderful holiday season, and I look forward to seeing and working with many of you in the coming year.
Until the next issue,
Judith A. Gundersen
Contact us to request a pdf file of the original article as it appeared in the print edition.
